Effects of the First Year of a 
Three-Year CGI Teacher 
ma ceyi---1(0)4}- 1 BL-\"-1 (0) olaalvals 
mceye|e-1a1e)amCle-\e(-\- Bane 
Student Achievement 


\ Multisite Cle ster-Randomized 1 Trial 


Research Report No. 2018-25 


The research and development reported here was supported by the Florida Department of 
Education, through Award Numbers 371-2355B-5C001, 371-2356B-6C001, and 371-2357B- 
7C004 to Florida State University. The opinions expressed are those of the authors and do not 
represent views of the Florida Department of Education. 


Suggested citation: Schoen, R. C., LaVenia, M., & Tazaz, A. M. (2018). Effects of the first year of a 
three-year CG! teacher professional development program on grades 3-5 student achievement: A 
multisite cluster-randomized trial (Research Report No. 2018-25). Tallahassee, FL: Learning 
Systems Institute, Florida State University. http://doi.org/10.33009/fsu.1562595 733 


Copyright 2018, Florida State University. All rights reserved. Requests for permission to use this 
test should be directed to Robert Schoen, rschoen@lsi.fsu.edu, FSU Learning Systems Institute, 
4600 University Center C, Tallahassee, FL, 32306 


Effects of the First Year of a Three-Year CGI Teacher Professonal 


Development Program on Grades 3-5 Student Achievement 
A Multisite Cluster-Randomized Trial 


Research Report No. 2018-25 


Robert C. Schoen 
Mark LaVenia 


Amanda M. Tazaz 


Florida Center for Research in Science, Technology, Engineering, and Mathematics (FCR-STEM) 
Learning Systems Institute 

Florida State University 

Tallahassee, FL 32306 

(850) 644-2570 


EFFECTS OF FIRST YEAR OF CGI 3-5 PROGRAM ON STUDENT ACHIEVEMENT 


Introduction 


Relative to the rest of the world, U.S. students’ performance in mathematics decreases 
between grade 4 and grade 8 (Mullis, Martin, Foy, & Hooper, 2016). The introduction of 
abstract concepts such as rational number has been theorized by members of the National 
Mathematics Advisory Panel (2008) as a likely cause of the dip in student performance in these 
grade levels, and proficiency with fractions has been identified as an important gatekeeper for 
success in Algebra (Booth & Newton, 2012; Empson & Levi, 2011; Siegler et al., 2012; Siegler, 
Thompson, & Schneider, 2011). 


Professional development for teachers is one possible strategy for improving student 
achievement. Regrettably, a recent trend toward rigorous evaluation of the impact of highly 
regarded teacher PD programs on student learning has yielded mostly null findings (Garet et al., 
2011; Garet et al., 2016; Jacob, Hill, & Corey, 2017; Jayanthi et al., 2017; Santagata et al., 2010). 
Ina review of the emerging evidence of the effects of teacher PD programs on student learning, 
Wilson (2013) concluded, “it is nearly impossible to isolate the effects of PD on student 
learning” (p. 312). 


One exception to the trend of null effects have been programs based on Cognitively Guided 
Instruction (CGI), which have had positive effects on student’s achievement (Gersten et al., 
2014). Until now, three extant studies employing experimental designs have reported positive 
effects on student achievement in mathematics. Carpenter et al. (1989) reported positive 
effects on first-grade students’ achievement on problem-solving tasks within the domain of 
whole-number addition and subtraction. Jacobs et al. (2007) reported positive effects on first- 
through sixth-grade students’ understanding of the equals sign and use of strategies that rely 
on algebraic relationships to solve equations. The third study—and the first one to be 
conducted by a third-party evaluator—reported results suggesting potentially positive effects 
on first-grade students’ problem-solving abilities (p <.10) and potentially negative effects on 
second-grade students computational abilities (o<.10) in the first year of the program (Schoen, 
et. al 2017) 


Although results concerning the impact of CGI PD on teachers and students are promising, 
much work is still needed in the area of research and evaluation in that CGI innovations are 
outpacing even the formidable body of CGl-related research and evaluation. Over the past 
thirty years, CGI has continually evolved and expanded. The developers of the CGI program 
have maintained a focus on increasing teachers’ understanding of children’s thinking as a 
means to improve teaching and learning, where problem-solving serves as the organizing focus 
for instruction (Carpenter et al., 1999; Carpenter & Franke, 2004). That focus has not changed. 
Whereas the original CGI program focused on addition and subtraction on whole numbers at 
the first-grade level (Carpenter et al., 1989), the content of CGI expanded through the 1990s to 
include multiplication and division on whole numbers and base-ten number concepts 
(Carpenter et al., 1999; Fennema et al., 1996). In the 2000s, the scope of the CGI program 
expanded further to focus on algebraic thinking (Carpenter, Franke, & Levi, 2003; Jacobs et al., 
2007). In the 2010s, Empson and Levi (2011) published frameworks to provide support for the 
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CGI approach in the domain of fractions and decimals, and Carpenter et al. (2016) published a 
similar book to support the CGI approach in the domain of counting and early number 
concepts. 


Purpose Statement 


The purpose of the present study was to estimate the impact of the first year of a three-year 
teacher professional development program designed for grades 3-5 mathematics teachers and 
their students. The study is guided by the following research question. 


What is the effect of the CG/ 3—5 program on grade 3, 4, and 5 student achievement as 
measured by the Spring 2016 Grades 3—5 Elementary Mathematics Student 
Assessment (EMSA)? 


Other mechanisms in the theory of change for the CGI program, such as the impact of the 
program on teacher knowledge and beliefs and the role of those factors as mediators of the 
effect of the program on student achievement, are addressed in the broader evaluation and 
research study and are beyond the scope of the present manuscript. 


Description of the CGI 3—5 Program 


The teachers in this study participated in the first year of a three-year CGI 3—5 professional 
development program. This particular model consisted of five consecutive seven-hour days 
during summer 2015; two consecutive six-hour days in fall 2015; and two consecutive days six- 
hour days in winter 2016. The CGI 3-5 program was designed and taught by certified CGI 
instructors at the Teachers Development Group under the direction of Linda Levi. Dr. Levi was 
the Director of CGI Initiatives at TDG and one of the co-authors of the three definitive CG! books 
as well as the CGI Guide for Workshop Leaders (Carpenter et al., 1999; Carpenter, Franke, & 
Levi, 2003; Empson & Levi, 2011; Fennema et al., 1999). 


The CGI 3-5 program was designed to focus teachers’ attention on their students’ 
mathematical thinking and to provide teachers with principled frameworks for understanding 
their students’ thinking. Teachers are introduced to two, complementary, researched-based 
frameworks during the PD workshops: 
e Problem Types Frameworks, which describe how the structure of a problem influences 
how children think about the mathematical concepts embedded in the problem; and 
e Solution Strategy Frameworks, which describe the developmental progressions of 
children’s mathematical thinking as determined by their strategies for solving problems 
within the problem-type framework. 


The frameworks addressed in the CGI 3—5 program describe children’s thinking about: fraction 


quantities; operations with fractions; whole number multiplication and division; and base-ten 
concepts for whole numbers and decimals. In addition to developing an understanding of these 
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frameworks, teachers are supported to use what they learn about students to drive 
instructional decisions. 


CGI 3—5 Program Theory of Change 


Figure 1 presents the CGI 3—5 program theory of change. The program aims to focus teachers’ 
attention on the details in students’ cognitive processes as they solve problems and encourages 
them to use what they learn about students to drive instructional decisions. It is thought to 
have a direct effect on teachers’ mathematical knowledge for teaching and beliefs about 
mathematics teaching and learning and an indirect effect on classroom instruction and student 
learning in mathematics. These effects build incrementally over the course of the school year 
(and across multiple school years for the multiyear program). 


Teachers learn about learning progressions related to student thinking and practice using these 
progressions to guide their instructional practice. During the classroom-embedded components 
of the professional-development workshops, teachers learn how to use problem solving as the 
organizing principle for instruction and orchestrate classroom discussions involving student 
sharing their various solution strategies and learning from one another. Teachers’ 
understanding of fractions and other mathematics concepts along with associated conventions 
of mathematical notation grows through in-depth study of children’s thinking processes. 


Classroom instruction following the CGI approach (Carpenter et al., 2015) uses a formative- 
assessment process wherein teachers observe students solving problems and explaining their 
thinking processes. Teachers use these observations to draw inference about students’ 
mathematical understanding. Students in CGI classrooms learn mathematics by engaging in 
problem solving, explaining their problem-solving strategies to the teacher and to their peers, 
and listening to various ways of solving problems. 


Methods 
Setting 


One hundred forty-nine grades 3-5 teachers—representing 32 schools, nine public school 
districts, and the geographic, socioeconomic, and cultural diversity of the central and northern 
regions of the state of Florida—participated in the study. Students in the analytic sample 
represented grades 3, 4, and 5. 


Enrollment 


Teachers were enrolled through an informed-consent process to voluntarily participate in the 
professional-development program and associated research study through a Web-based survey 
deployed in spring 2015. To qualify for enrollment in the study, the applicants needed to be 
expecting to teach in a participating school district, and their expected teaching assignment for 
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SY 2015-16 needed to involve teaching mathematics to intermediate-grades students. At the 
start of SY 2015-16, teachers distributed an approved letter from the principal investigator to 
parents and guardians in accordance with the consent procedure approved by the participating 
school districts and the Institutional Review Board at Florida State University. 


Randomization 


Blocking on schools, individual teachers were assigned at random to the intervention group or 
the wait-list comparison group. Teachers assigned to the comparison condition participated in 
business-as-usual professional development and mathematics instruction during the study year 
and were invited to participate in the intervention program starting the summer after the 
student and teacher posttests were completed. 


A total of 34 randomization blocks were used. The additional 2 blocks (greater than the number 
of schools) are attributable to two teachers being part of a rolling assignment process (to meet 
the target sample size); both teachers were late applicants in schools where there were 
teachers who were already randomized into the study. The number of participating teachers 
per school ranged from three to eight. The decision rule for the blocking procedure was, if the 
number of participating teachers at a school was three to five, two teachers were randomly 
assigned to treatment and the others to control; if the number of participating teachers at a 
school was six or more, three were randomly assigned to treatment and the others to control. 
For the two rolling-assignment blocks, teachers had a 50 percent chance of assignment to 
treatment. This approach resulted in 69 of the 149 teachers being assigned to treatment. 

As described in the data analysis section below, the covariate set for the impact analyses 
included dummy variables for randomization blocks, which is one of three methods considered 
acceptable by the What Works Clearinghouse (WWC, 2017) for accounting for different 
assignment probabilities. 


Data Sources 


Baseline student achievement was measured using the 2015 Fall Grades 3—5 Elementary 
Mathematics Student Assessment (EMSA) test during the first two weeks of the school year 
(Schoen, Anderson, Champagne, & Bauduin, 2018). Student posttest mathematics achievement 
was measured in May 2016 using the Spring 2016 3—5 EMSA test (Schoen, Anderson, & 
Bauduin, 2018). The tests were designed to assess student achievement in understanding 
fraction quantities and operations involving whole numbers and fractions. 


Teachers provided consent to participate and completed pretest assessments during April— 
June, 2015, prior to participation in the summer training. Teachers completed posttest 
assessments in April-May, 2016. Students completed pretest assessments during the first two 
weeks of the 2015-16 school year. Students completed posttest assessments during an 
assessment window spanning late-April to late-May in 2016. 
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Sample Attrition and Cluster Baseline Equivalence 


All impact estimates were conducted as an intent-to-treat analysis of randomized teachers and 
their respective students. We used a joiners model for the student sample, where all students 
with consent to participate who were enrolled in a randomized teacher’s classroom at the time 
of post-testing were eligible for inclusion in the analytic sample. The analytic sample included 
those students eligible for inclusion in the analytic sample who completed the outcome 
measure administered spring 2016. Teacher-level attrition was evaluated based on randomized 
teachers with students who met the inclusion criteria for the analytic sample relative to all 
randomized teachers. Student-level attrition was evaluated based on students in the analytic 
sample relative to all students on randomized teachers’ spring 2016 class rosters (excluding 
students within attriting teacher clusters), which were provided by the school districts’ and 
based on their administrative data. 


Table 1 presents attrition rates for teacher clusters and students. The overall and differential 
rates for teacher clusters were 32.9% and 11.6%, respectively. According to WWC (2017) 
guidelines, this exceeds the boundary for acceptable threat of bias due to attrition. However, 
the overall and differential rates of 9.2% and 4.4%, respectively, for students are below the 
cautious WWC boundary for acceptable threat of bias due to attrition. 


Equivalence of clusters at baseline was established using fall 2015 student achievement data for 
all students who were enrolled in a non-attriting teacher’s classroom spring 2016. The pretest 
and the posttest measures correlated at r= .76, p< .001. As presented in Table 2, using 
classroom means and standard deviations for the calculation, a group difference of g = .21 was 
calculated; using individual student scores and standard deviations for the calculation, a group 
difference of g = .17 was calculated. Parallel information is provided in Table A1 in Appendix A. 
Regardless of calculation method, results indicate the group difference at baseline was within 
acceptable range, providing statistical adjustment for pretest in the analysis, according to WWC 
guidelines (2017). 


Data Analysis 


Data were fit to a two-level model involving students nested in teachers with fixed effects for 
n—-1 randomization blocks. Table 3 presents a description of the analytic models as fit using 
Moplus 8 (Muthén & Muthén, 1998-2017). Model 1 includes binary indicator covariates for 
student grade level, treatment condition, and randomization block. Model 2 adds a continuous 
covariate for pretest. All models used the Mplus MLR maximum likelihood with robust standard 
errors estimator. All impact analyses constituted a second stage of analysis, where the initial 
stage involved calculation of IRT-based scores for the pre- and posttest student achievement 
measures. The saved student-ability estimates were treated as observed variables in the impact 
analyses. 


For all impact analyses, we used a full information maximum likelihood (FIML) approach, as 
described by Muthén, Muthén, and Asparouhov (2016), to avoid dropping cases missing data 
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on the pretest measure. FIML is one of several WWC-accepted approaches for handling missing 
data (WWC, 2017). 


Sensitivity and Verification 


Additional analyses were conducted using a complete-case sample with respect to student pre- 
and posttest data. Models 1 and 2 were fit to the complete-case sample, as were exploratory 
models to test whether effects of treatment were moderated by student grade level or baseline 
performance on the student pretest. Using the same data, these models were fit independently 
by the projects’ external evaluator using Stata v. 15.1 (StataCorp, 2017) and Rv. 3.4.4(R 
Development Core Team, 2014) Ime4 package (Bates, Maechler, Bolker, & Walker, 2015). 


Results 
Main Effects 


The results of the statistical models designed to estimate the main effect of the program on 
student achievement are provided in Table 4. Using FIML with the full analytic sample, we 
estimated an effect for treatment in Model 2 of g = 0.18 (p = .007). Results of the sensitivity 
analysis using the complete-case sample are provided in Table 5. For the complete-case 
analysis, we estimated an effect for treatment in Model 2 of g = 0.13 (p = .030). The unadjusted 
standard deviations and analytic sample sizes are provided in the Appendix in Table A2. 


Moderation Analyses 


We did not find any statistically significant interactions between treatment and grade level or 
treatment and baseline achievement scores. 


Sensitivity Analysis 


Results from our analyses and those conducted by the external evaluators were practically 
identical and had no meaningful discrepancies apart from what would be expected by different 
software routines. Further, the inference drawn from the complete-case analysis did not differ 
from the inference drawn from the results of the model that used FIML, suggesting the results 
are robust to the influence of missing data on sample composition. 


Discussion 
The CGI 3-5 PD program was found to significantly impact student mathematics achievement (p 


=.007) in the first year of the three-year program. The point estimate of the main effect of the 
CGI program on student achievement (g = .18) was greater than many of the most effective 
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teacher professional-development programs that have been subjected to rigorous evaluation 
(Kennedy, 2016). 


Ina recent evaluation brief, Garet et al. (2016) discussed three teacher PD programs that did 
not translate to increases in student learning. They concluded that teacher PD can improve 
teachers’ content knowledge and practice, to some degree, but “that the field does not yet fully 
understand how to ensure that teacher PD leads to measurable improvements in student 
learning” (p. 11). The findings reported in the present study provide a counterexample to this 
narrative and suggest that the field can learn from the design and implementation of the CGI 3- 
5 program. 


Some of the leading ideas for improving student learning in fractions currently focus on the use 
of linear representations and relative magnitude of fractions, including identifying fractions as 
points on the number line (Booth, Newton, & Twiss-Garrity, 2014; Lewis & Perry, 2017, 
Torbeyns, Schneider, Xin, & Siegler, 2015). CGI 3-5 PD takes a different approach from these 
other initiatives. Rather than introducing fractions through part-whole models or as points on a 
number line, fractions are introduced as an extension of whole number division through Equal 
Sharing Problems, such as: 


If there are 3 brownies, and 4 people want to share them equally, how much brownie 
should each person get? 


In the CGI approach to instruction, students are supported in developing their intuitive 
strategies to solve mathematics problems involving fractions and, in the process, develop a 
strong understanding of fractional quantities and the properties of operations as they related 
to operations with fractions. Students are not shown how to solve these problems. 

Instead, they produce their own representations of fractional quantities in the process of 
solving each problem. Word problems are also used as vehicles for introducing fractions, 
operations on fractions and whole numbers, and equivalence relations. Word problems, and 
equations associated with these problems, continue to be the primary tools for teaching 
fractions concepts and operations after such concepts are introduced. For more information on 
this approach, see Empson and Levi (2011). 


CGI programs have garnered considerable praise and attention from mathematics educators 
and mathematics teacher educators since the program’s inception. These new results provide 
further evidence that the CGI approach to professional development may extend to higher 
grade levels. A more complete analysis of the theory of change and implementation of the CGI 
3-5 program will be necessary to yield additional insight into the generalizable and scalable 
components of the design and implementation that resulted in the positive impact on students. 
Future work should involve (a) replication of this study, (b) a longer-term study to evaluate the 
impact of the full three-year program, and (c) a more complete analysis of the impact of the 
program on teachers’ knowledge, beliefs, and behaviors, including classroom-instruction data, 
and the role of those factors in mediating the effect of the program on students. 
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| Intervention | Direct Effects on Teachers | Indirect Effects on Students | 


Cognitive Outcomes Instructional Practice 
Outcomes 
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Contextual factors: Coaching and other school-based support for teacher learning and implementation; principal support for 
enactment of CGI principles; District Administrators and parent support; flexibility in adjusting the instructional plan based 
on students’ needs. 


Figure 1. Theory of change for the CG/ 3-5 program. 
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Table 1. Teacher Cluster Attrition and Students Representativeness 


Attrition/ 
Analytic sample N Reference sample N Representativeness 
Treatment Control Total Treatment Control Total Overall Differential 
Randomly assigned teacher clusters 
Teachers 42 58 100 69 80 149 32.89% 11.63% 
Students (excluding those within attriting teacher clusters) 
Students 957 1,249 2,206 1,083 1,347 2,430 9.22% 4.36% 
Note. Analytic sample is defined as all students with data on the outcome measure. 
Table 2. Baseline Equivalence of Clusters 
Small-sample 
bias correction 
Baseline M Hedges' g standard deviation calculation calculation 
Mi Me ni SD?; Nc SD" SDpooled W g 
Using classroom means and standard deviations 
0.193 0.002 43 0.979 52 0.743 0.922 0.992 0.205 
Using individual student scores and standard deviations 
0.321 0.114 844 1599 963 1.304 1.201 1,807 1.000 0.172 
Note. Subscript i and c refer to the intervention and control group, respectively. 
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Table 3. Description of Analytic Models as Fit using Mplus 


Model Construct/Variable Variable description Mplus modeling particulars 
M1 Student characteristics 
Grade 4 Binary variable indicating Modeled at Level-1, as fixed 
student was in Grade 4 effect. Variance is estimated at 
Level-1. 
Grade 5 Binary variable indicating Modeled at Level-1, as fixed 
student was in Grade 5 effect. Variance is estimated at 
Level-1. 
Group characteristics 
Block Vector of binary indicators for Modeled at Level-2, as fixed 
randomized block (school) 1D, — effect. 
using Block 35 as the reference 
category. 
Workshop Binary variable indicating Modeled at Level-2, as fixed 
group was assigned to effect. 
workshop condition. 
M2 Student characteristics All 


variables in M1 plus: 
Pretest 


Continuous variable for 
student mathematics 
achievement fall 2015 pretest, 
vertically scaled for grades 3-5 


Modeled at Level-1, as fixed 
effect. Variance is estimated at 
Level-1. 


Note. The Mplus MLR maximum likelihood with robust standard errors estimator is used for both models. The 
specification that variance for independent variables is estimation at Level-1 applies to only analyses using FIML to 
handle missing data; estimation of variance at Level-1 for independent variables is not necessary for complete case 


analysis. 
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Table 4. Impact Analyses with the Full Sample using FIML to Handle Missing Data 


Model 1 


Model 2 


Coeff 
(SE) Effect size p 


Coeff 
(SE) Effect size p 


Fixed effects 
Within classroom 


Grade 4 0.423 _ < .001 
(0.098) 

Grade 5 1.069 _ < .001 
(0.119) 

Pretest 


Between classroom 


Block? = = — 
Workshop 0.189 0.17 .041 
(0.092) 
Intercept -0.294 _ .032 
(0.137) 
Variance components 
Within classroom 0.800 (0.035) <.001 
Between classrooms 0.107 (0.026) < .001 


-0.022 = 798 
(0.086) 
0.022 = 838 
(0.106) 
0.653 = <.001 
(0.025) 
0.202 0.18 .007 
(0.075) 
0.005 = 968 
(0.129) 
0.450 (0.016) <.001 
0.058 (0.015) <.001 


Note. Student N = 2,206; Teacher N = 100. Reported estimates are unstandardized. Only the effect size for the 


Workshop treatment variable is presented. 


*Block indicates the vector of n—1 randomization blocks. Parameters for Block are not presented for visual 


simplicity. 


Coefficients reported at the within-level for random slopes are the random intercepts estimated at the between- 


level. 
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Table 5. /mpact Analyses with the Complete-Case Sample 


Model 1 Model 2 
Coeff Coeff 
(SE) Effect size p (SE) Effect size p 
Fixed effects 
Within classroom 
Grade 4 0.430 — < .001 -0.162 — .040 
(0.106) (0.079) 
Grade 5 1.060 — < .001 -0.237 — .011 
(0.129) (0.094) 
Pretest 0.722 — < .001 
(0.024) 
Between classroom 
Block? — _ — — — — 
Workshop 0.161 0.14 .106 0.147 0.13 .030 
(0.099) (0.068) 
Intercept -0.225 _ .150 0.111 _ .386 
(0.156) (0.128) 
Variance components 
Within classroom 0.781 (0.037) <.001 0.452 (0.016) < .001 
Between classrooms 0.106 (0.027) <.001 0.038 (0.009) < .001 


Note. Student N = 1,683; Teacher N = 92. Reported estimates are unstandardized. Only the effect size for the 
Workshop treatment variable is presented. 
*Block indicates the vector of n—1 randomization blocks. Parameters for Block are not presented for visual 


simplicity. 
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Appendix A 


Table A1. Baseline Equivalence Information for Complete-Case Analysis (n = 1,683) reported in 
Table 5 


Intervention group Comparison group Total 
Student sample size 770 913 1,683 
Student-level EMSA 0.386304 0.092678 
Pretest Mean 
Student-level EMSA 1.269690 1.139906 
Pretest SD 


Table A2. Unadjusted SDs for Analytic Sample for Complete-Case Analysis in Table 5 


Analytic-sample 


unadjusted standard Analytic-sample size 
HLM coefficient deviation (n) 
Intervention group 1.127458 770 
Comparison group 0.147 1.101253 913 
Total 1,683 


